Statistical Mechanics of Learning
نویسنده
چکیده
We review the application of statistical mechanics methods to the study of online learning of a drifting concept in the limit of large systems. The model where a feed-forward network learns from examples generated by a time dependent teacher of the same architecture is analyzed. The best possible generalization ability is determined exactly, through the use of a variational method. The constructive variational method also suggests a learning algorithm. It depends, however, on some unavailable quantities, such as the present performance of the student. The construction of estimators for these quantities permits the implementation of a very eeective, highly adaptive algorithm. Several other algorithms are also studied for comparison with the optimal bound and the adaptive algorithm, for diierent types of time evolution of the rule.
منابع مشابه
Statistical mechanics of learning in the presence of outliers
Abstract. Using methods of statistical mechanics, we analyse the effect of outliers on the supervised learning of a classification problem. The learning strategy aims at selecting informative examples and discarding outliers. We compare two algorithms which perform the selection either in a soft or a hard way. When the fraction of outliers grows large, the estimation errors undergo a first-orde...
متن کاملRetarded learning: rigorous results from statistical mechanics.
We study learning of probability distributions characterized by an unknown symmetry direction. Based on an entropic performance measure and the variational method of statistical mechanics we develop exact upper and lower bounds on the scaled critical number of examples below which learning of the direction is impossible. The asymptotic tightness of the bounds suggests an asymptotically optimal ...
متن کاملNonextensive statistical mechanics for hybrid learning of neural networks
This paper introduces a new hybrid approach for learning systems that builds on the theory of nonextensive statistical mechanics. The proposed learning scheme uses only the sign of the gradient, and combines adaptive stepsize local searches with global search steps that make use of an annealing schedule inspired from nonextensive statistics, as proposed by Tsallis. The performance of the hybrid...
متن کاملTransients and Asymptotics of Natural Gradient Learning
We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.
متن کاملStatistical Mechanics of Mutual Learning with a Latent Teacher
We propose a mutual learning with a latent teacher within the framework of on-line learning, and have analyzed its dynamical behavior through the statistical mechanics method. The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. A teacher is not used in the mutual learning, so...
متن کامل